Vehicle vision system with smart camera video output

ABSTRACT

A vehicular vision system includes at least one color camera disposed at a vehicle and having an image sensor operable to capture image data. A first system on chip (SoC) includes an image signal processor that receives captured image data and converts the received image data to converted data that is in a format suitable for machine vision processing. A second system on chip (SoC) receives captured image data and communicates display data to a display disposed in the vehicle, with the display data being in a format suitable for display of video images at the display. At startup of the vehicle, video images derived from the display data are displayed by the display within a time period following startup of the vehicle and machine vision data processing of converted data does not commence until after the display time period has elapsed following startup of the vehicle.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of U.S. patent applicationSer. No. 15/421,483, filed Feb. 1, 2017, now U.S. Pat. No. 11,433,809,which claims the filing benefits of U.S. provisional application Ser.No. 62/290,168, filed Feb. 2, 2016, which is hereby incorporated hereinby reference in its entirety.

FIELD OF THE INVENTION

The present invention relates generally to a vehicle vision system for avehicle and, more particularly, to a vehicle vision system that utilizesone or more cameras.

BACKGROUND OF THE INVENTION

Use of imaging sensors in vehicle imaging systems is common and known.Examples of such known systems are described in U.S. Pat. Nos.5,949,331; 5,670,935 and/or 5,550,677, which are hereby incorporatedherein by reference in their entireties.

SUMMARY OF THE INVENTION

The present invention provides a driver assistance system or visionsystem or imaging system for a vehicle that utilizes one or more camerasto capture image data representative of images exterior of the vehicle.The camera includes a first chip having a processor that receivescaptured image data (such as captured by an imager or imaging sensor ofthe camera) and processes the received captured image data for machinevision, wherein, responsive to image processing of the received capturedimage data, the first chip or processor generates an output for a driverassistance system of the vehicle. The camera includes a second chip orprocessor that receives the captured image data and that communicatesvideo images to a display for viewing by the driver of the vehicleduring normal operation of the vehicle.

The present invention thus provides a smart camera that can quickly(upon initial vehicle start up) provide video images for display to thedriver of the vehicle, without having to wait for the machine visionprocessor to warm up and process the image data captured by the camera.The smart camera provides an output responsive to image processing ofthe captured image data for a vehicle driver assistance system, and alsoprovides an output for displaying video images at a display of thevehicle.

These and other objects, advantages, purposes and features of thepresent invention will become apparent upon review of the followingspecification in conjunction with the drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a plan view of a vehicle with a vision system thatincorporates cameras in accordance with the present invention;

FIG. 2 is a block diagram of a known vision system; and

FIG. 3 is a block diagram of the vision system of the present invention.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

A vehicle vision system and/or driver assist system and/or objectdetection system and/or alert system operates to capture images exteriorof the vehicle and may process the captured image data to display imagesand to detect objects at or near the vehicle and in the predicted pathof the vehicle, such as to assist a driver of the vehicle in maneuveringthe vehicle in a rearward direction. The vision system includes an imageprocessor or image processing system that is operable to receive imagedata from one or more cameras and provide an output to a display devicefor displaying images representative of the captured image data.Optionally, the vision system may provide display, such as a rearviewdisplay or a top down or bird's eye or surround view display or thelike.

Referring now to the drawings and the illustrative embodiments depictedtherein, a vehicle 10 includes an imaging system or vision system 12that includes at least one exterior facing imaging sensor or camera,such as a rearward facing imaging sensor or camera 14 a (and the systemmay optionally include multiple exterior facing imaging sensors orcameras, such as a forwardly facing camera 14 b at the front (or at thewindshield) of the vehicle, and a sidewardly/rearwardly facing camera 14c, 14 d at respective sides of the vehicle), which captures imagesexterior of the vehicle, with the camera having a lens for focusingimages at or onto an imaging array or imaging plane or imager of thecamera (FIG. 1 ). Optionally, a forward viewing camera may be disposedat the windshield of the vehicle and view through the windshield andforward of the vehicle, such as for a machine vision system (such as fortraffic sign recognition, headlamp control, pedestrian detection,collision avoidance, lane marker detection and/or the like). The visionsystem 12 includes a control or electronic control unit (ECU) orprocessor 18 that is operable to process image data captured by thecamera or cameras and may detect objects or the like and/or providedisplayed images at a display device 16 for viewing by the driver of thevehicle (although shown in FIG. 1 as being part of or incorporated in orat an interior rearview mirror assembly 20 of the vehicle, the controland/or the display device may be disposed elsewhere at or in thevehicle). The data transfer or signal communication from the camera tothe ECU may comprise any suitable data or communication link, such as avehicle network bus or the like of the equipped vehicle.

The present invention provides a vision system that has a reduced videostart-up time for smart camera applications. The system creates a highreliability video pipeline independent of a machine vision pipeline.Since video is considered a fundamental feature to be provided by avehicle camera, the independence from complex software associated withmachine vision and image processing creates inherent reliability for thevideo pipeline. The increased reliability in hardware allows for anenhanced safety critical system design. The present invention alsoimproves the quality of the machine vision algorithm by avoiding thedigital image manipulation that is applied to enhance human perceptionin a viewing application.

The image sensor of a vision system captures image data to create animage of the environment, and outputs the captured image data (such asshown in FIG. 2 ). The image format could be many of the known formats(such as, for example, Bayer, RGB, YUV and/or the like). The system onchip (SoC) is responsible for machine vision. The SoC includes an ISPblock (shown as ISP) to convert the raw image into a format suitable formachine vision algorithms as well as for display.

FIG. 2 shows the common implementation block diagram of a camera for avehicle vision system. The input image, image processing (including bothfor viewing and for machine vision), and output image are in series. Thedrawback of such system is that there is a delay in video startup forthe vehicle driver due to delay caused by software boot-up andinitialization of the system on chip (SoC) 3. The SoC software iscomplicated and lengthy and thus the boot-time is long, however thevehicle driver expects the video from the vehicle camera 1 to beviewable (such as at a display 5) as soon as the vehicle 7 is switchedon. There is a NHSTA requirement for video start-up time as well inFMVSS 111.

The output of the system on chip 3 includes the results of machinevision algorithms and status information for the vehicle (such as, forexample, objects detected, location of pedestrians detected and anywarnings) on a communication bus 6. The communication bus 6 may be anyphysical layer common in automotive industry (such as, for example, aCAN bus, LIN bus, Ethernet, I2C encoded on LVDS and/or the like).

The SoC 3 is also responsible for video output to vehicle display. Itincludes the ISP block which converts the raw video 2 to an outputformat suitable for display. This output to the display could be eitherNTSC or LVDS 4 or any other video format which can be sent to vehicledisplay.

For better human perception of displayed images, the system may applydigital sharpening, higher contrast and more saturated color in imagesignal processing (ISP) of captured image data for generating videoimages for the display. However, these changes sometimes pose challengesto machine vision algorithms that process the changed image data.

As shown in FIG. 3 , the system and smart camera of the presentinvention provides two chips 23, 26 for a smart camera 15 of an advanceddriver assistance system (ADAS). The image sensor 14 captures image dataand creates an image of the environment, and outputs it in an imageformat 22. The image format 22 may comprise any known formats (such as,for example, Bayer, RGB, YUV and/or the like). The first system on chip(SoC) 23 is responsible for machine vision and includes an independentISP block (shown as ISP_1) to convert the image data into formatsuitable for machine vision algorithms (where the first SoC may performfurther processing for machine vision systems or may communicate theconverted image data to an image processor of a driver assistance systemthat processes the communicated converted image data (such as for objectdetection or lane marker detection or traffic sign recognition or thelike). The second SoC 26 communicates the unconverted image data fordisplay by the display screen for viewing by the driver of the vehicle.

The output 27 of the first SoC 23 includes the results of machine visionalgorithms and status information for the vehicle (such as, for example,objects detected, location of pedestrians detected and any warnings).

The second SoC 26 is responsible for video output 24 to the vehicledisplay 25. The SoC 26 includes an ISP block (shown as ISP_2) tailoredto convert raw video 22 to an output format suitable for display. Theoutput of the second SoC 26 may comprise NTSC video or LVDS or anyformat which can be sent to vehicle display. Because the output of thesecond SoC is based on image data that is not processed for machinevision (by the first SoC 23), the second SoC may also apply digitalsharpening, higher contrast and/or more saturated color in the imagesignal processing (ISP) of the captured image data for generatingenhanced video images for the display.

The image format (in its as-captured format) for the image datacommunicated by the second SoC is uncompressed data directly output fromthe imager's array (such as a color Bayer pixelated array or the like)and may be any suitable format. The first SoC is responsible for machinevision (which processes converted data), while the second SoCcommunicates unprocessed image data (that may be converted to colorspace such as via, for example, RGB, YUV and/or the like) to the displaysystem for display of video images. Both processors may be disposed ator may be part of the camera or may be remote from the camera, such asat a control of the vehicle or system.

Thus, the present invention provides two processors or chips thatreceive the raw image data captured by the imaging sensor or imagingarray or imager of the camera, with one processor converting and/orprocessing the image data for machine vision functions (such as objectdetection or the like) and communicating an output to one or morevehicle systems (such as a backup assist system or adaptive cruisecontrol system or headlamp control system or lane departure warningsystem or the like), and with the other processor providing video imagesfor display at a display unit of the vehicle. Thus, the system andcamera of the present invention provides for rapid display of videoimages when the vehicle is first turned on (such as when the ignition isinitially turned to on or start), since the video images are notprocessed by the machine vision processor (which may still be warming upafter video images are being displayed on the display screen). Thepresent invention provides enhanced early display of video images, whichis important at the onset of a reversing maneuver, so the driver canquickly see the displayed video images without having to wait forprocessing algorithms to “wake up” and start processing the raw imagedata to detect objects or the like.

The cameras may comprise any suitable camera or sensor. Optionally, thecamera may comprise a “smart camera” that includes the imaging sensorarray and associated circuitry and image processing circuitry andelectrical connectors and the like as part of a camera module, such asby utilizing aspects of the vision systems described in InternationalPublication Nos. WO 2013/081984 and/or WO 2013/081985, which are herebyincorporated herein by reference in their entireties.

The system includes an image processor operable to process image datacaptured by the camera or cameras, such as for detecting objects orother vehicles or pedestrians or the like in the field of view of one ormore of the cameras. For example, the image processor (SoC 23) maycomprise an image processing chip selected from the EYEQ family of imageprocessing chips available from Mobileye Vision Technologies Ltd. ofJerusalem, Israel, and may include object detection software (such asthe types described in U.S. Pat. Nos. 7,855,755; 7,720,580 and/or7,038,577, which are hereby incorporated herein by reference in theirentireties), and may analyze image data to detect vehicles and/or otherobjects. Responsive to such image processing, and when an object orother vehicle is detected, the system may generate an alert to thedriver of the vehicle and/or may generate an overlay at the displayedimage to highlight or enhance display of the detected object or vehicle,in order to enhance the driver's awareness of the detected object orvehicle or hazardous condition during a driving maneuver of the equippedvehicle.

The vehicle may include any type of sensor or sensors, such as imagingsensors or radar sensors or lidar sensors or ladar sensors or ultrasonicsensors or the like. The imaging sensor or camera may capture image datafor image processing and may comprise any suitable camera or sensingdevice, such as, for example, a two dimensional array of a plurality ofphotosensor elements arranged in at least 640 columns and 480 rows (atleast a 640×480 imaging array, such as a megapixel imaging array or thelike), with a respective lens focusing images onto respective portionsof the array. The photosensor array may comprise a plurality ofphotosensor elements arranged in a photosensor array having rows andcolumns. Preferably, the imaging array has at least 300,000 photosensorelements or pixels, more preferably at least 500,000 photosensorelements or pixels and more preferably at least 1 million photosensorelements or pixels. The imaging array may capture color image data, suchas via spectral filtering at the array, such as via an RGB (red, greenand blue) filter or via a red/red complement filter or such as via anRCC (red, clear, clear) filter or the like. The logic and controlcircuit of the imaging sensor may function in any known manner, and theimage processing and algorithmic processing may comprise any suitablemeans for processing the images and/or image data.

For example, the vision system and/or processing and/or camera and/orcircuitry may utilize aspects described in U.S. Pat. Nos. 8,694,224;7,005,974; 5,760,962; 5,877,897; 5,796,094; 5,949,331; 6,302,545;6,396,397; 6,498,620; 6,523,964; 6,611,202; 6,201,642; 6,690,268;6,717,610; 6,757,109; 6,802,617; 6,806,452; 6,822,563; 6,891,563;6,946,978; 7,859,565; 5,550,677; 5,670,935; 7,881,496; 7,720,580;7,038,577; 6,882,287; 5,929,786 and/or 5,786,772, which are all herebyincorporated herein by reference in their entireties. The system maycommunicate with other communication systems via any suitable means,such as by utilizing aspects of the systems described in InternationalPublication Nos. WO 2010/144900; WO 2013/043661 and/or WO 2013/081985,and/or U.S. Publication No. US-2012-0062743, which are herebyincorporated herein by reference in their entireties.

Optionally, the vision system may include a display for displayingimages captured by one or more of the imaging sensors for viewing by thedriver of the vehicle while the driver is normally operating thevehicle. Optionally, for example, the vision system may include a videodisplay device disposed at or in the interior rearview mirror assemblyof the vehicle, such as by utilizing aspects of the video mirror displaysystems described in U.S. Pat. Nos. 5,530,240; 6,329,925; 7,855,755;7,626,749; 7,581,859; 7,446,650; 7,370,983; 7,338,177; 7,274,501;7,255,451; 7,195,381; 7,184,190; 5,668,663; 5,724,187 and/or 6,690,268,and/or U.S. Publication Nos. US-2006-0061008; US-2006-0050018, and/orUS-2012-0162427, which are hereby incorporated herein by reference intheir entireties. The display may utilize aspects of the display systemsdescribed in U.S. Pat. No. 8,427,751 and/or U.S. Publication Nos.US-2014-0333729; US-2014-0139676; US-2015-0092042; US-2015-0232030and/or US-2016-0209647, which are all hereby incorporated herein byreference in their entireties. Optionally, the vision system (utilizingthe forward facing camera and a rearward facing camera and other camerasdisposed at the vehicle with exterior fields of view) may be part of ormay provide a display of a top-down view or birds-eye view system of thevehicle or a surround view at the vehicle, such as by utilizing aspectsof the vision systems described in International Publication Nos. WO2010/099416; WO 2011/028686; WO 2012/075250; WO 2013/019795; WO2012/075250; WO 2012/145822; WO 2013/081985; WO 2013/086249 and/or WO2013/109869, and/or U.S. Publication No. US-2012-0162427, which arehereby incorporated herein by reference in their entireties.

Changes and modifications in the specifically described embodiments canbe carried out without departing from the principles of the invention,which is intended to be limited only by the scope of the appendedclaims, as interpreted according to the principles of patent lawincluding the doctrine of equivalents.

The invention claimed is:
 1. A vehicular vision system, the vehicularvision system comprising: a color camera disposed at a vehicle equippedwith the vehicular vision system, the color camera viewing exterior ofthe vehicle, the color camera comprising an image sensor, wherein thecolor camera is operable to capture image data; wherein the image sensorhas at least one million photosensor elements arranged in rows andcolumns, and wherein the color camera captures color image data viaspectral filtering at the image sensor; a display disposed in thevehicle and operable to display video images for viewing by a driver ofthe vehicle during operation of the vehicle; an image data processingchip that comprises a first system on a chip (SoC) that includes animage signal processor that receives image data captured by the colorcamera and converts the received image data to converted data that is ina format suitable for machine vision processing; wherein the image dataprocessing chip processes the converted data using machine vision dataprocessing and, responsive to the machine vision data processing of theconverted data, generates an output indicative of detection of apedestrian present in a field of view of the color camera; wherein theoutput indicative of detection of the pedestrian is provided to avehicle system of the vehicle; a video processing chip that comprises asecond system on a chip (SoC) that receives image data captured by thecolor camera and communicates display data to the display, and whereinthe display data is derived from received image data captured by thecolor camera, and wherein the display data is in a format suitable fordisplay of video images at the display; wherein the display displaysvideo images derived from the display data communicated from the videoprocessing chip; and wherein, at startup of the vehicle, video imagesderived from the display data communicated from the video processingchip are displayed by the display within a time period following startupof the vehicle and machine vision data processing of converted data atthe image data processing chip does not commence until after the displaytime period has elapsed following startup of the vehicle.
 2. Thevehicular vision system of claim 1, wherein the display data comprisesvideo images derived from image data as captured by the color camera. 3.The vehicular vision system of claim 2, wherein the video images arecommunicated to the display upon startup of the vehicle while the firstSoC is booting up and not yet able to process the converted data.
 4. Thevehicular vision system of claim 1, wherein the video processing chipconverts the received image data captured by the color camera to theformat suitable for display of video images at the display.
 5. Thevehicular vision system of claim 4, wherein the display datacommunicated by the video processing chip is derived from captured imagedata that is not processed by the image data processing chip.
 6. Thevehicular vision system of claim 5, wherein the video processing chipapplies digital sharpening, enhanced contrast and enhanced saturatedcolor to the captured image data for generating enhanced video imagesfor the display.
 7. The vehicular vision system of claim 1, wherein thecolor camera disposed at the equipped vehicle is one of a plurality ofcolor cameras disposed at the vehicle.
 8. The vehicular vision system ofclaim 1, wherein the color camera comprises the image data processingchip and the video processing chip.
 9. The vehicular vision system ofclaim 1, wherein the color camera comprises a rear backup cameradisposed at a rear portion of the vehicle.
 10. The vehicular visionsystem of claim 1, wherein the color camera comprises a forward viewingcamera disposed at a windshield of the vehicle and viewing forward ofthe vehicle through the windshield.
 11. The vehicular vision system ofclaim 1, wherein the display data is communicated by the videoprocessing chip to the display via low-voltage differential signaling(LVDS).
 12. The vehicular vision system of claim 1, wherein the outputof the image data processing chip is provided to the vehicle system ofthe vehicle via a communication bus of the vehicle.
 13. The vehicularvision system of claim 1, wherein the vehicle system of the vehiclecomprises a backup assist system of the vehicle.
 14. The vehicularvision system of claim 1, wherein the vehicle system of the vehiclecomprises an adaptive cruise control system of the vehicle.
 15. Avehicular vision system, the vehicular vision system comprising: a colorcamera disposed at a vehicle equipped with the vehicular vision system,the color camera viewing exterior of the vehicle, the color cameracomprising an image sensor, wherein the color camera is operable tocapture image data; wherein the image sensor has at least one millionphotosensor elements arranged in rows and columns, and wherein the colorcamera captures color image data via spectral filtering at the imagesensor; a display disposed in the vehicle and operable to display videoimages for viewing by a driver of the vehicle during operation of thevehicle; an image data processing chip that comprises a first system ona chip (SoC) that includes an image signal processor that receives imagedata captured by the color camera and converts the received image datato converted data that is in a format suitable for machine visionprocessing; wherein the image data processing chip processes theconverted data using machine vision data processing and, responsive tothe machine vision data processing of the converted data, generates anoutput indicative of detection of a pedestrian present in a field ofview of the color camera; wherein the output indicative of detection ofthe pedestrian is provided to a vehicle system of the vehicle via acommunication bus of the vehicle; a video processing chip that comprisesa second system on a chip (SoC) that receives image data captured by thecolor camera and communicates display data to the display, and whereinthe display data is derived from received image data captured by thecolor camera, and wherein the display data is in a format suitable fordisplay of video images at the display; wherein the display data iscommunicated by the video processing chip to the display via low-voltagedifferential signaling (LVDS); wherein the display displays video imagesderived from the display data communicated from the video processingchip; and wherein, at startup of the vehicle, video images derived fromthe display data communicated from the video processing chip aredisplayed by the display within a time period following startup of thevehicle and machine vision data processing of converted data at theimage data processing chip does not commence until after the displaytime period has elapsed following startup of the vehicle.
 16. Thevehicular vision system of claim 15, wherein the display datacommunicated by the video processing chip is derived from captured imagedata that is not processed by the image data processing chip.
 17. Thevehicular vision system of claim 15, wherein the video processing chipapplies digital sharpening, enhanced contrast and enhanced saturatedcolor to the captured image data for generating enhanced video imagesfor the display.
 18. The vehicular vision system of claim 15, whereinthe color camera disposed at the equipped vehicle is one of a pluralityof color cameras disposed at the vehicle.
 19. The vehicular visionsystem of claim 15, wherein the color camera comprises a rear backupcamera disposed at a rear portion of the vehicle.
 20. The vehicularvision system of claim 15, wherein the color camera comprises a forwardviewing camera disposed at a windshield of the vehicle and viewingforward of the vehicle through the windshield.
 21. The vehicular visionsystem of claim 15, wherein the vehicle system of the vehicle comprisesa backup assist system of the vehicle.
 22. The vehicular vision systemof claim 15, wherein the vehicle system of the vehicle comprises anadaptive cruise control system of the vehicle.
 23. A vehicular visionsystem, the vehicular vision system comprising: a color camera disposedat a vehicle equipped with the vehicular vision system, the color cameraviewing exterior of the vehicle, the color camera comprising an imagesensor, wherein the color camera is operable to capture image data;wherein the image sensor has at least one million photosensor elementsarranged in rows and columns, and wherein the color camera capturescolor image data via spectral filtering at the image sensor; a displaydisposed in the vehicle and operable to display video images for viewingby a driver of the vehicle during operation of the vehicle; an imagedata processing chip that comprises a first system on a chip (SoC) thatincludes an image signal processor that receives image data captured bythe color camera and converts the received image data to converted datathat is in a format suitable for machine vision processing; wherein theimage data processing chip processes the converted data using machinevision data processing and, responsive to the machine vision dataprocessing of the converted data, generates an output indicative ofdetection of a pedestrian present in a field of view of the colorcamera; wherein the output indicative of detection of the pedestrian isprovided to a vehicle system of the vehicle; a video processing chipthat comprises a second system on a chip (SoC) that receives image datacaptured by the color camera and converts the received image data todisplay data that is in a format suitable for display of video images atthe display, and wherein the display data is communicated to thedisplay; wherein the display displays video images derived from thedisplay data communicated from the video processing chip; wherein, atstartup of the vehicle, video images derived from the display datacommunicated from the video processing chip are displayed by the displaywithin a time period following startup of the vehicle and machine visiondata processing of converted data at the image data processing chip doesnot commence until after the display time period has elapsed followingstartup of the vehicle; and wherein the display data is communicated tothe display upon startup of the vehicle while the first SoC is bootingup and not yet able to process the converted data.
 24. The vehicularvision system of claim 23, wherein the display data communicated by thevideo processing chip is derived from captured image data that is notprocessed by the image data processing chip.
 25. The vehicular visionsystem of claim 23, wherein the video processing chip applies digitalsharpening, enhanced contrast and enhanced saturated color to thecaptured image data for generating enhanced video images for thedisplay.
 26. The vehicular vision system of claim 23, wherein the colorcamera disposed at the equipped vehicle is one of a plurality of colorcameras disposed at the vehicle.
 27. The vehicular vision system ofclaim 23, wherein the color camera comprises a rear backup cameradisposed at a rear portion of the vehicle.
 28. The vehicular visionsystem of claim 23, wherein the color camera comprises a forward viewingcamera disposed at a windshield of the vehicle and viewing forward ofthe vehicle through the windshield.
 29. The vehicular vision system ofclaim 23, wherein the vehicle system of the vehicle comprises a backupassist system of the vehicle.
 30. The vehicular vision system of claim23, wherein the vehicle system of the vehicle comprises an adaptivecruise control system of the vehicle.